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Based  on  our  experience  with  software  system  development  and  our  expertise  with  collaborative  numerical 
solvers  (simulation-engines),  our  objective  in  this  projects  is  to  develop  a  computational  methodology  and 
software  environment  that  supports  collaborative  solvers.  We  utilized  the  funding  in  this  project  to  purchase  a 
high-end  Sun  Enterprise  multiprocessor  and  three  Dell  workstations  with  wireless  hubs.  These  machines  are 
used,  along  with  another  Sun  multiprocessor,  to  perform  multiprocessor-cluster  experiments.  Our  focus  is  on 
runtime  support  for  collaboration  between  solvers,  and  runtime  support  for  user-level  collaboration  between 
analysts  (simulation  engine  developers  and/or  users)  who  interact  with  solvers  on  neighboring  sub-domains. 

At  the  simulation-engine  level,  collaborative  support  implies  efficient  mechanisms  for  managing  distributed 
threads  and  synchronization  (API,  functionality  and  efficiency  issues)  and  scalability  (threads,  and 
ommunication).  At  the  user-level,  collaborative  support  implies  efficient  communication  protocols 
(multiprotocol  support,  realtime,  multiway,  low-latency  and  high-throughput  communication)  that  provide  all 
the  requisite  efficiency  and  functionality  for  user-level  collaborative  interactions. 

Borrowing  from  our  experiences  with  CLAM  (multithreaded  protocols)  and  ParaSol  (speculative,  migrant- 
threads  based  distributed  simulation),  we  propose  an  architecture  for  which  the  major  portion  of  our  research 
deals  with  issues  at  the  lowest  layer.  Here,  the  Ariadne  and  Arachne  threads  systems  offer  threads  functionality, 
and  the  Clam  communication  substem  offers  highly  efficient  communication  functionality.  The  kernel  layer 
exports  Ariadne  or  Arachne  threads  to  the  solver  layer,  to  new  or  legacy  solvers  like  Ellpack,  provides  support 
for  threads  scheduling  and  process  synchronization  via  speculative  executions.  We  are  currently  experimenting 
with  a  methodology  in  which  application  layer  code  can  be  written  without  use  of  host  ids,  explicit 
send/receive  or  processor  synchronization  primitives,  with  appropriate  help  from  the  solver  layer,  object  layer 
and  kernel  layer. 

We  have  had  considerable  success  with  migrant-threads  and  object  proxies,  and  in  implementing  optimistic  and 
speculative  executions  in  the  ParaSol  simulation  system.  We  are  currently  using  this  methodology  to 
synchronize  processors,  buidling  functionality  into  the  kernel  layer.  The  object  layer  is  an  application- 
independent  layer,  but  requires  object  definitions  for  every  interface  (boundary)  in  the  problem  space.  Because 
these  definitions  depend  on  the  data  (problem)  and  not  the  solver,  we  are  attempting  to  define  them  by  class 
libraries  which  represent  different  boundary  geometries.  With  this,  the  object  layer  is  made  completely 
independent  of  the  data  structures  used  by  legacy  systems,  and  also  provides  for  object  definitions  in  new 
multithreaded  solvers.  We  initially  plan  to  offer  users  a  “bind_subdomain(A,  hostid)”  primitive  that,  at  the  start 
of  execution,  statically  binds  subdomain  A  to  a  solver  running  on  proc  (hostid).  Since  the  object  layer  will 
be  equipped  with  an  object  location  mechanism,  users  are  relieved  of  the  responsibility  of  programming  in 
terms  of  processor  “ids”.  Once  “bind_subdomain(A,  hostid)”  runs  to  establish  the  object-processor  map,  any 
invocation  of  object  A  by  a  thread  automatically  and  transparently  migrates  the  thread  to  processor  hostid 
which  hosts  subdomain  A.  We  have  used  this  system  very  effectively  in  both  Ariadne  and  also  in  ParaSol. 

Because  a  collaborative  solver  must  iterate  and  communicate  with  neighboring  solvers  in  a  chaotic  way, 
communication  is  highly  irregular.  To  enable  such  unpredictable  communication  patterns  and  yet  enhance 
communication  performance  (i.e.,  by  eliminating  high  polling  costs  which  involve  several  layers  of  a 
communication  protocol  like  TCP/IP),  and  to  make  speculative  executions  possible,  all  (interface)  objects  in  the 
object  layer  must  provide  SAVE  and  RESTORE  methods,  so  that  they  can  be  transparently  checkpointed  at 
appropriate  times  that  are  indexed  by  iteration  indices.  These  checkpointed  states  will  be  (transparently) 
used  to  restore  objects,  whenever  necessary,  during  speculative  computations. 


The  SOLVER  LAYER  provides  for  either  legacy  or  new  solvers.  Using  envelopes,  modular  legacy  solvers  like 
EllPack  can  be  implemented  as  three  fat  threads:  (l).~an  “input”  thread  that  a  user  can  interact  with  for 
dynamic  parameter  modification,  (2).~an  “output”  thread  which  offers  dynamic  display  of  simulation  results, 
and  (3).~a  “solver”  thread  that  iterates  over  its  subdomain,  given  boundary  values.  New  solvers  can  be  fully 
designed  in  terms  of  an  arbitrary  number  of  threads,  to  maximize  the  potential  benefits  of  design  simplicity, 
concurrency  on  multiprocessors,  and  performance.  For  example,  even  with  as  few  as  three  threads  for  legacy 
solvers  like  EllPack,  Ariadne’s  time-slicing  feature  enables  separate  actions  to  be  performed  concurrently: 
exeution,  output  visualization  and  runtime  modification  of  parameters.  Depending  on  the  legacy  code, 
further  concurrency  and  object-definition  support  for  checkpointing  and  restoration  (to  offer  more  flexibility  at 
the  application-level)  may  be  also  be  possible. 
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